{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Labeling: Tail Sets\n",
    "\n",
    "## Abstract\n",
    "\n",
    "Tail set labels are a classification labeling technique introduced in the following paper: \"[Huerta, R., Corbacho, F. and\n",
    "Elkan, C., 2013. Nonlinear support vector machines can systematically identify stocks with high and low future returns.\n",
    "Algorithmic Finance, 2(1), pp.45-58.](https://content.iospress.com/download/algorithmic-finance/af016?id=algorithmic-finance%2Faf016)\n",
    "\n",
    "A tail set is defined to be a group of assets whose volatility-adjusted price change is in the highest or lowest\n",
    "quantile, for example the highest or lowest 5%.\n",
    "\n",
    "A classification model is then fit using these labels to determine which stocks to buy and sell, for a long / short\n",
    "portfolio.\n",
    "\n",
    "## How it works\n",
    "\n",
    "We label the y variable using the tail set labeling technique, which makes up the positive and negative (1, -1) classes\n",
    "of the training data. The original paper investigates the performance of 3 types of metrics on which the tail sets are\n",
    "built:\n",
    "\n",
    "1. Real returns\n",
    "2. Residual alpha after regression on the sector index\n",
    "3. Volatility-adjusted returns\n",
    "\n",
    "For our particular implementation, we have focused on the volatility-adjusted returns.\n",
    "\n",
    "An input DataFrame of prices is converted to returns, which can have volatility adjustment applied. The formula for volatility-adjusted return is:\n",
    "\n",
    "$$r(t - t', t) = \\frac{R(t-t',t)}{vol(t)}$$\n",
    "\n",
    "We provide two implementations for estimations of volatility, first the exponential moving average of the mean absolute returns, and second the traditional standard deviation. The paper suggests a 180 day window period. \n",
    "\n",
    "The volatility adjusted return of each stock is assigned to a quantile relative to other returns in the row i.e. same timestamp. The top and bottom quantiles are then labeled as the positive and negative classes, respectively.\n",
    "\n",
    "## How to use these labels in practice?\n",
    "\n",
    "The tail set labels from the code above returns the names of the assets which should be labeled with a positive or\n",
    "negative label. It's important to note that the model you  would develop is a many to one model, in that it has many\n",
    "x variables and only one y variable. The model is a binary classifier.\n",
    "\n",
    "The model is trained on the training data and then used to score every security in the test data (on a given day).\n",
    "Example: On December 1st 2019, the strategy needs to rebalance its positions, we score all 100 securities in our tradable\n",
    "universe and then rank the outputs in a top-down fashion. We form a long / short portfolio by going long the top 10\n",
    "stocks and short the bottom 10 (equally weighted). We then hold the position to the next rebalance date.\n",
    "\n",
    "---\n",
    "## Examples of use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import yfinance as yf\n",
    "\n",
    "from mlfinlab.labeling import TailSetLabels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[*********************100%***********************]  20 of 20 completed\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BABA</th>\n",
       "      <th>PFE</th>\n",
       "      <th>AAPL</th>\n",
       "      <th>MSFT</th>\n",
       "      <th>CVX</th>\n",
       "      <th>F</th>\n",
       "      <th>KO</th>\n",
       "      <th>FB</th>\n",
       "      <th>GE</th>\n",
       "      <th>JPM</th>\n",
       "      <th>COST</th>\n",
       "      <th>GPS</th>\n",
       "      <th>ACI</th>\n",
       "      <th>AMD</th>\n",
       "      <th>ZM</th>\n",
       "      <th>WFC</th>\n",
       "      <th>TWTR</th>\n",
       "      <th>SYY</th>\n",
       "      <th>NVDA</th>\n",
       "      <th>CCL</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2019-01-22</th>\n",
       "      <td>152.149994</td>\n",
       "      <td>39.946537</td>\n",
       "      <td>150.266403</td>\n",
       "      <td>103.568062</td>\n",
       "      <td>105.162872</td>\n",
       "      <td>7.837517</td>\n",
       "      <td>45.439438</td>\n",
       "      <td>147.570007</td>\n",
       "      <td>8.272988</td>\n",
       "      <td>98.963676</td>\n",
       "      <td>209.413116</td>\n",
       "      <td>22.838694</td>\n",
       "      <td>5100.0</td>\n",
       "      <td>19.760000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>46.484818</td>\n",
       "      <td>32.250000</td>\n",
       "      <td>60.661575</td>\n",
       "      <td>148.035126</td>\n",
       "      <td>51.750248</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-01-23</th>\n",
       "      <td>152.029999</td>\n",
       "      <td>39.842587</td>\n",
       "      <td>150.874130</td>\n",
       "      <td>104.577469</td>\n",
       "      <td>104.273567</td>\n",
       "      <td>7.689988</td>\n",
       "      <td>45.963150</td>\n",
       "      <td>144.300003</td>\n",
       "      <td>8.339860</td>\n",
       "      <td>98.713730</td>\n",
       "      <td>209.107468</td>\n",
       "      <td>23.241623</td>\n",
       "      <td>5100.0</td>\n",
       "      <td>19.799999</td>\n",
       "      <td>NaN</td>\n",
       "      <td>46.727222</td>\n",
       "      <td>30.969999</td>\n",
       "      <td>60.884239</td>\n",
       "      <td>148.552521</td>\n",
       "      <td>51.512947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-01-24</th>\n",
       "      <td>155.860001</td>\n",
       "      <td>38.699097</td>\n",
       "      <td>149.678253</td>\n",
       "      <td>104.077667</td>\n",
       "      <td>106.258125</td>\n",
       "      <td>7.929724</td>\n",
       "      <td>45.410866</td>\n",
       "      <td>145.830002</td>\n",
       "      <td>8.387626</td>\n",
       "      <td>98.771400</td>\n",
       "      <td>207.352509</td>\n",
       "      <td>23.122576</td>\n",
       "      <td>5100.0</td>\n",
       "      <td>20.850000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>46.596699</td>\n",
       "      <td>31.610001</td>\n",
       "      <td>60.535717</td>\n",
       "      <td>157.060287</td>\n",
       "      <td>52.224846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-01-25</th>\n",
       "      <td>159.210007</td>\n",
       "      <td>38.406132</td>\n",
       "      <td>154.638153</td>\n",
       "      <td>105.028282</td>\n",
       "      <td>105.986641</td>\n",
       "      <td>8.169458</td>\n",
       "      <td>45.106155</td>\n",
       "      <td>149.009995</td>\n",
       "      <td>8.750644</td>\n",
       "      <td>99.396294</td>\n",
       "      <td>206.129929</td>\n",
       "      <td>23.516348</td>\n",
       "      <td>5100.0</td>\n",
       "      <td>21.930000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>46.736546</td>\n",
       "      <td>32.900002</td>\n",
       "      <td>60.041988</td>\n",
       "      <td>159.358887</td>\n",
       "      <td>52.689953</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2019-01-28</th>\n",
       "      <td>158.919998</td>\n",
       "      <td>37.357147</td>\n",
       "      <td>153.207047</td>\n",
       "      <td>102.980049</td>\n",
       "      <td>105.003731</td>\n",
       "      <td>7.985046</td>\n",
       "      <td>44.915710</td>\n",
       "      <td>147.470001</td>\n",
       "      <td>8.530922</td>\n",
       "      <td>99.867378</td>\n",
       "      <td>207.806046</td>\n",
       "      <td>23.598763</td>\n",
       "      <td>5100.0</td>\n",
       "      <td>20.180000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>46.447533</td>\n",
       "      <td>33.130001</td>\n",
       "      <td>60.284012</td>\n",
       "      <td>137.328262</td>\n",
       "      <td>53.534740</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  BABA        PFE        AAPL        MSFT         CVX  \\\n",
       "Date                                                                    \n",
       "2019-01-22  152.149994  39.946537  150.266403  103.568062  105.162872   \n",
       "2019-01-23  152.029999  39.842587  150.874130  104.577469  104.273567   \n",
       "2019-01-24  155.860001  38.699097  149.678253  104.077667  106.258125   \n",
       "2019-01-25  159.210007  38.406132  154.638153  105.028282  105.986641   \n",
       "2019-01-28  158.919998  37.357147  153.207047  102.980049  105.003731   \n",
       "\n",
       "                   F         KO          FB        GE        JPM        COST  \\\n",
       "Date                                                                           \n",
       "2019-01-22  7.837517  45.439438  147.570007  8.272988  98.963676  209.413116   \n",
       "2019-01-23  7.689988  45.963150  144.300003  8.339860  98.713730  209.107468   \n",
       "2019-01-24  7.929724  45.410866  145.830002  8.387626  98.771400  207.352509   \n",
       "2019-01-25  8.169458  45.106155  149.009995  8.750644  99.396294  206.129929   \n",
       "2019-01-28  7.985046  44.915710  147.470001  8.530922  99.867378  207.806046   \n",
       "\n",
       "                  GPS     ACI        AMD  ZM        WFC       TWTR        SYY  \\\n",
       "Date                                                                            \n",
       "2019-01-22  22.838694  5100.0  19.760000 NaN  46.484818  32.250000  60.661575   \n",
       "2019-01-23  23.241623  5100.0  19.799999 NaN  46.727222  30.969999  60.884239   \n",
       "2019-01-24  23.122576  5100.0  20.850000 NaN  46.596699  31.610001  60.535717   \n",
       "2019-01-25  23.516348  5100.0  21.930000 NaN  46.736546  32.900002  60.041988   \n",
       "2019-01-28  23.598763  5100.0  20.180000 NaN  46.447533  33.130001  60.284012   \n",
       "\n",
       "                  NVDA        CCL  \n",
       "Date                               \n",
       "2019-01-22  148.035126  51.750248  \n",
       "2019-01-23  148.552521  51.512947  \n",
       "2019-01-24  157.060287  52.224846  \n",
       "2019-01-25  159.358887  52.689953  \n",
       "2019-01-28  137.328262  53.534740  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load price data for 20 stocks\n",
    "tickers = \"AAPL MSFT COST PFE SYY F GE BABA AMD CCL ZM FB WFC JPM NVDA CVX TWTR ACI GPS KO\"\n",
    "\n",
    "data = yf.download(tickers, start=\"2019-01-20\", end=\"2020-05-25\", group_by=\"ticker\")\n",
    "data = data.loc[:, (slice(None), 'Adj Close')]\n",
    "data.columns = data.columns.droplevel(1)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create tail set labels with mean absolute deviation as the volatility adjustment.\n",
    "labels = TailSetLabels(data, n_bins=10, vol_adj='mean_abs_dev', window=180)\n",
    "pos_set, neg_set, matrix_set = labels.get_tail_sets()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date\n",
       "2020-01-06      [GPS, ZM]\n",
       "2020-01-07     [ZM, TWTR]\n",
       "2020-01-08    [MSFT, SYY]\n",
       "2020-01-09     [KO, COST]\n",
       "2020-01-10     [PFE, GPS]\n",
       "dtype: object"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the positive set, of the top 10% returns for each day.\n",
    "pos_set.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date\n",
       "2020-01-06    [WFC, CCL]\n",
       "2020-01-07    [CVX, JPM]\n",
       "2020-01-08     [CVX, GE]\n",
       "2020-01-09    [PFE, GPS]\n",
       "2020-01-10     [GE, JPM]\n",
       "dtype: object"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the negative set, of the lowest 10% returns for each day.\n",
    "neg_set.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BABA</th>\n",
       "      <th>PFE</th>\n",
       "      <th>AAPL</th>\n",
       "      <th>MSFT</th>\n",
       "      <th>CVX</th>\n",
       "      <th>F</th>\n",
       "      <th>KO</th>\n",
       "      <th>FB</th>\n",
       "      <th>GE</th>\n",
       "      <th>JPM</th>\n",
       "      <th>COST</th>\n",
       "      <th>GPS</th>\n",
       "      <th>ACI</th>\n",
       "      <th>AMD</th>\n",
       "      <th>ZM</th>\n",
       "      <th>WFC</th>\n",
       "      <th>TWTR</th>\n",
       "      <th>SYY</th>\n",
       "      <th>NVDA</th>\n",
       "      <th>CCL</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2020-01-06</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-07</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-08</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-09</th>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-10</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            BABA  PFE  AAPL  MSFT  CVX  F  KO  FB  GE  JPM  COST  GPS  ACI  \\\n",
       "Date                                                                         \n",
       "2020-01-06     0    0     0     0    0  0   0   0   0    0     0    1    0   \n",
       "2020-01-07     0    0     0     0   -1  0   0   0   0   -1     0    0    0   \n",
       "2020-01-08     0    0     0     1   -1  0   0   0  -1    0     0    0    0   \n",
       "2020-01-09     0   -1     0     0    0  0   1   0   0    0     1   -1    0   \n",
       "2020-01-10     0    1     0     0    0  0   0   0  -1   -1     0    1    0   \n",
       "\n",
       "            AMD  ZM  WFC  TWTR  SYY  NVDA  CCL  \n",
       "Date                                            \n",
       "2020-01-06    0   1   -1     0    0     0   -1  \n",
       "2020-01-07    0   1    0     1    0     0    0  \n",
       "2020-01-08    0   0    0     0    1     0    0  \n",
       "2020-01-09    0   0    0     0    0     0    0  \n",
       "2020-01-10    0   0    0     0    0     0    0  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# All labels for the day.\n",
    "matrix_set.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BABA</th>\n",
       "      <th>PFE</th>\n",
       "      <th>AAPL</th>\n",
       "      <th>MSFT</th>\n",
       "      <th>CVX</th>\n",
       "      <th>F</th>\n",
       "      <th>KO</th>\n",
       "      <th>FB</th>\n",
       "      <th>GE</th>\n",
       "      <th>JPM</th>\n",
       "      <th>COST</th>\n",
       "      <th>GPS</th>\n",
       "      <th>ACI</th>\n",
       "      <th>AMD</th>\n",
       "      <th>ZM</th>\n",
       "      <th>WFC</th>\n",
       "      <th>TWTR</th>\n",
       "      <th>SYY</th>\n",
       "      <th>NVDA</th>\n",
       "      <th>CCL</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2020-01-06</th>\n",
       "      <td>-0.121334</td>\n",
       "      <td>-0.163835</td>\n",
       "      <td>0.745082</td>\n",
       "      <td>0.311705</td>\n",
       "      <td>-0.423720</td>\n",
       "      <td>-0.510353</td>\n",
       "      <td>-0.056495</td>\n",
       "      <td>1.680486</td>\n",
       "      <td>0.905581</td>\n",
       "      <td>-0.098610</td>\n",
       "      <td>0.039689</td>\n",
       "      <td>2.396394</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.227461</td>\n",
       "      <td>1.907438</td>\n",
       "      <td>-0.650362</td>\n",
       "      <td>0.259407</td>\n",
       "      <td>-0.217696</td>\n",
       "      <td>0.266647</td>\n",
       "      <td>-2.423721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-07</th>\n",
       "      <td>0.336044</td>\n",
       "      <td>-0.430076</td>\n",
       "      <td>-0.445724</td>\n",
       "      <td>-1.104470</td>\n",
       "      <td>-1.592236</td>\n",
       "      <td>0.917607</td>\n",
       "      <td>-1.188172</td>\n",
       "      <td>0.196681</td>\n",
       "      <td>-0.481040</td>\n",
       "      <td>-2.095482</td>\n",
       "      <td>-0.230710</td>\n",
       "      <td>-0.021452</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.153854</td>\n",
       "      <td>0.959547</td>\n",
       "      <td>-0.901826</td>\n",
       "      <td>1.892687</td>\n",
       "      <td>-1.346047</td>\n",
       "      <td>0.769067</td>\n",
       "      <td>0.246498</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-08</th>\n",
       "      <td>0.126608</td>\n",
       "      <td>1.022932</td>\n",
       "      <td>1.499143</td>\n",
       "      <td>1.883782</td>\n",
       "      <td>-1.415710</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.286338</td>\n",
       "      <td>0.918880</td>\n",
       "      <td>-0.595934</td>\n",
       "      <td>0.950251</td>\n",
       "      <td>1.652935</td>\n",
       "      <td>0.115619</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.467444</td>\n",
       "      <td>0.391693</td>\n",
       "      <td>0.331650</td>\n",
       "      <td>1.048754</td>\n",
       "      <td>1.894668</td>\n",
       "      <td>0.121107</td>\n",
       "      <td>0.379514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-09</th>\n",
       "      <td>1.276731</td>\n",
       "      <td>-0.563110</td>\n",
       "      <td>1.950353</td>\n",
       "      <td>1.471037</td>\n",
       "      <td>-0.201126</td>\n",
       "      <td>0.103904</td>\n",
       "      <td>2.743687</td>\n",
       "      <td>1.289600</td>\n",
       "      <td>-0.165231</td>\n",
       "      <td>0.448849</td>\n",
       "      <td>2.271257</td>\n",
       "      <td>-1.860596</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.255253</td>\n",
       "      <td>0.042490</td>\n",
       "      <td>-0.188380</td>\n",
       "      <td>0.348897</td>\n",
       "      <td>0.196094</td>\n",
       "      <td>0.708613</td>\n",
       "      <td>0.723627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-01-10</th>\n",
       "      <td>0.686095</td>\n",
       "      <td>1.952353</td>\n",
       "      <td>0.211654</td>\n",
       "      <td>-0.552650</td>\n",
       "      <td>-1.136807</td>\n",
       "      <td>-0.105099</td>\n",
       "      <td>0.524137</td>\n",
       "      <td>-0.100989</td>\n",
       "      <td>-1.331385</td>\n",
       "      <td>-1.230105</td>\n",
       "      <td>-1.041752</td>\n",
       "      <td>1.082667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.879143</td>\n",
       "      <td>0.286838</td>\n",
       "      <td>-0.486054</td>\n",
       "      <td>-0.907804</td>\n",
       "      <td>0.570522</td>\n",
       "      <td>0.349014</td>\n",
       "      <td>-0.578200</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                BABA       PFE      AAPL      MSFT       CVX         F  \\\n",
       "Date                                                                     \n",
       "2020-01-06 -0.121334 -0.163835  0.745082  0.311705 -0.423720 -0.510353   \n",
       "2020-01-07  0.336044 -0.430076 -0.445724 -1.104470 -1.592236  0.917607   \n",
       "2020-01-08  0.126608  1.022932  1.499143  1.883782 -1.415710  0.000000   \n",
       "2020-01-09  1.276731 -0.563110  1.950353  1.471037 -0.201126  0.103904   \n",
       "2020-01-10  0.686095  1.952353  0.211654 -0.552650 -1.136807 -0.105099   \n",
       "\n",
       "                  KO        FB        GE       JPM      COST       GPS  ACI  \\\n",
       "Date                                                                          \n",
       "2020-01-06 -0.056495  1.680486  0.905581 -0.098610  0.039689  2.396394  0.0   \n",
       "2020-01-07 -1.188172  0.196681 -0.481040 -2.095482 -0.230710 -0.021452  0.0   \n",
       "2020-01-08  0.286338  0.918880 -0.595934  0.950251  1.652935  0.115619  0.0   \n",
       "2020-01-09  2.743687  1.289600 -0.165231  0.448849  2.271257 -1.860596  0.0   \n",
       "2020-01-10  0.524137 -0.100989 -1.331385 -1.230105 -1.041752  1.082667  0.0   \n",
       "\n",
       "                 AMD        ZM       WFC      TWTR       SYY      NVDA  \\\n",
       "Date                                                                     \n",
       "2020-01-06 -0.227461  1.907438 -0.650362  0.259407 -0.217696  0.266647   \n",
       "2020-01-07 -0.153854  0.959547 -0.901826  1.892687 -1.346047  0.769067   \n",
       "2020-01-08 -0.467444  0.391693  0.331650  1.048754  1.894668  0.121107   \n",
       "2020-01-09  1.255253  0.042490 -0.188380  0.348897  0.196094  0.708613   \n",
       "2020-01-10 -0.879143  0.286838 -0.486054 -0.907804  0.570522  0.349014   \n",
       "\n",
       "                 CCL  \n",
       "Date                  \n",
       "2020-01-06 -2.423721  \n",
       "2020-01-07  0.246498  \n",
       "2020-01-08  0.379514  \n",
       "2020-01-09  0.723627  \n",
       "2020-01-10 -0.578200  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# See the numerical returns.\n",
    "labels.vol_adj_rets.dropna().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error Handling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Errors will be raised if inputs are invalid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n_bins exceeds the number of stocks!\n",
      "If vol_adj is not None, window must be int.\n",
      "Length of price data must be greater than the window.\n"
     ]
    }
   ],
   "source": [
    "# If number of bins is greater than the width of the price data i.e. exceeds the number of stocks.\n",
    "try:\n",
    "    TailSetLabels(data[:100], n_bins=50)\n",
    "except Exception as exc:\n",
    "    print(exc)\n",
    "\n",
    "# If window is either not an int or too small.\n",
    "try:\n",
    "    TailSetLabels(data[:100], n_bins=10, vol_adj='stdev', window='str')\n",
    "except Exception as exc:\n",
    "    print(exc)\n",
    "try:\n",
    "    TailSetLabels(data[:100], n_bins=10, vol_adj='stdev', window=200)\n",
    "except Exception as exc:\n",
    "    print(exc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook presents the tail sets labeling method. This method is useful in identifying outliers in the returns for a group of stocks during a given day. The user chooses the number of quantiles, and the top and bottom quantiles are labeled as the positive and negative tail sets, respectively. This method can be used in training data for classification. A strategy can be adopted of going long the predicted positive tail set and short the negative one."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Huerta, R., Corbacho, F. and Elkan, C., 2013. Nonlinear support vector machines can systematically identify stocks with high and low future returns. Algorithmic Finance, 2(1), pp.45-58."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
